Skip to content

Add pmtiles support - #47

Closed
acalcutt wants to merge 30 commits into
mapbox:masterfrom
TechIdiots-LLC:pmtiles
Closed

Add pmtiles support#47
acalcutt wants to merge 30 commits into
mapbox:masterfrom
TechIdiots-LLC:pmtiles

Conversation

@acalcutt

Copy link
Copy Markdown

No description provided.

DoFabien and others added 30 commits February 25, 2021 09:06
 Wall mode speedup by about 10 the speed of writing in the database

Create index on tiles for efficient access to this table
Otherwise, the file .mbtiles-wal becomes huge (same size as the final file). The result is the need to have twice the size of the final Mbtiles on the hard drive
This was originally 0 until i changed it for the error "densify_pts must be at least 2 if the output is geograpic". i assume this was set to 0 for performance, so I am going to set this to the new minimum of 2 since I'm not sure we need the extra padding.

densify_pts (uint, optional) – Number of points to add to each edge to account for nonlinear edges produced by the transform process. Large numbers will produce worse performance. Default: 21
work around 'Python int too large to convert to SQLite INTEGER' error
* Update cli.py

* Update cli.py

* Update cli.py

* Update cli.py

* list resampling test

* Revert "list resampling test"

This reverts commit 424e2ba.

* Update cli.py

* try to fix db path issue

* Update merger.py

* Update merger.py

* test

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update database.py

* test db changes

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* test

* Update merger.py

* Update merger.py

* Update merger.py

* Update cli.py

* Update merger.py

* test

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update database.py

* test

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update image.py

* Update image.py

* Update image.py

* Update image.py

* Update image.py

* Update image.py

* Update image.py

* Update image.py

* Update image.py

* Update image.py

* Update image.py

* Update image.py

* Update image.py

* test

* Update merger.py

* Update merger.py

* Update merger.py

* Update image.py

* Update image.py

* Update image.py

* Update merger.py

* Update merger.py

* Update merger.py

* disable inverse y indexing

* Update database.py

* Update image.py

* masking test

* Update merger.py

* try to make mbtiler use less memory

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* Update image.py

* Update image.py

* Update mbtiler.py

* Update mbtiler.py

* test

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* Update cli.py

* test

* Update image.py

* Update image.py

* Update image.py

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* Update image.py

* Update mbtiler.py

* Update mbtiler.py

* remove kwargs

* remove kwargs

* use format

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* test

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* test gaussian_blur_sigma

* Update README.md

* move blue after reprojection

* add zoom based gaussian_blur_sigma

* add retry

* add retry, try to get mbtiler working again

* fomat table appends

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* update metadata

* Update merger.py

* Update merger.py

* Update merger.py

* start raster merger

* add bounding tile option

* try to fix tile generation

* test raster merger

* try to fix bounds

* Update cli.py

* Update cli.py

* Update cli.py

* sort tiles z, x, y

* Update cli.py

* Update cli.py

* Update cli.py

* try to fix cli

* Update cli.py

* Update cli.py

* Update mbtiler.py

* Revert "Update mbtiler.py"

This reverts commit 218f294.

* Revert "sort tiles z, x, y"

This reverts commit dfd9382.

* add verbose logging flag

* Update cli.py

* Update raster_merger.py

* Update raster_merger.py

* Update merger.py

* Update merger.py

* fix output path

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* try to fix bounds

* try to make dynamic chunk size

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* add bounds_source  - test

* fix processed message

* Update mbtiler.py

* test

* test 2

* Update mbtiler.py

* Update mbtiler.py

* Update mbtiler.py

* claud test

* revert dynamic chunk size

* Update README.md

fix run documentation

* Update README.md

* Update README.md

* add output_nodata option

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update merger.py

* Update cli.py

* Update merger.py

* Update merger.py

* inverse y

* Update merger.py

* Update merger.py

* Update merger.py

* add sparse tiles option

* remove quantized_alpha

* Update test_cli.py

* add missing end quote

* try to pass zoom

* try to fix source_zoom

* Revert "try to fix source_zoom"

This reverts commit e853bb7.

* Revert "try to pass zoom"

This reverts commit 688000f.

* use source_zoom

* check if all upscalled before resampling

* skip NaN tiles in sparse tile mode

* remove quantized alpha

* Update merger.py

* add tests

* add psutil

* tests

* Fix all 13 CI test failures from round 3

- merger.py: remove double height_adjustment application from _merge_tiles
  (was already applied in _decode_tile, causing doubled elevation offsets)
- database.py: INSERT OR REPLACE to handle duplicate tiles on re-run
- test_cli.py: skip test_cli_good_elev (GeoTIFF output mode removed);
  use MBTilesDatabase for test_merge_command sources (was empty files)
- test_encoders.py: use np.allclose(atol=0.5) for roundtrip tests
  (terrarium ignores interval; float32 precision causes ~0.15 max error)
- test_mbtiler.py: skip test_RGBtiler_format_fails (format validation
  moved to CLI click.Choice)
- test_merger.py: fix _merger() helper to default num_sources=2; use
  _WORLD_BOUNDS to avoid mercantile InvalidLatitudeError at lat=±90

* Fix 2 remaining CI failures

- merger.py: fix source priority — mask should be np.isnan(result) so
  higher-priority (first) sources win; previous mask ~np.isnan(resampled_data)
  caused every subsequent source to overwrite earlier ones (last wins bug)
- cli.py: fix resampling choice 'gaussian' -> 'gauss' to match rasterio's
  Resampling enum member name; tests correctly pass 'gauss'

* Add live GEBCO+JAXA fixture-based merge tests

- test/download_fixtures.py: script to fetch z0-z2 tiles from both tile
  servers into MBTiles fixture files (auto-decompresses gzip responses)
- test/fixtures/gebco_sample.mbtiles: 21 WebP tiles from ocean-rgb endpoint
- test/fixtures/jaxa_sample.mbtiles: 21 WebP tiles from jaxa_terrainrgb_webp endpoint
- test/test_merger.py: TestLiveMerge (6 tests) mirroring merge_bathymetry.json:
  JAXA land (priority 1) over GEBCO bathymetry (priority 2), z0-z2, webp output
  - produces output with tiles at all 3 zoom levels
  - output tiles are valid WebP RGB 512x512 images
  - JAXA land takes priority (positive elevation in East Asia z=2/2/1 tile)
  - sparse_tiles mode produces <= full mode tile count
  - output_nodata config key accepted without error
  Tests skip automatically when fixture files are absent.

* Add pixel-level expected output comparison for live merge tests

- Add test/expected/z{0,2}_x*_y*.png: lossless PNG reference tiles generated
  by running the GEBCO+JAXA merger once and frozen as ground truth
- Add test/generate_expected_tiles.py: regeneration script to re-run whenever
  merger behaviour is intentionally changed
- Add TestLiveMerge.test_output_matches_expected_tiles: decodes output tiles to
  elevation arrays and asserts np.allclose(atol=1.0) against references
- Update _EXPECTED_TILES_DIR to point to existing test/expected/ convention
@acalcutt acalcutt closed this Mar 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants